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Another round of PISA reporting has come and gone and the world is still spinning. So is the 
media. For the classroom teacher, generally, today is another day where they will teach and 
children and youth will learn. So do the PISA results impact on a day in the life of a classroom 
teacher? | would suggest that they do but the degree of impact is probably related to where you 
live. Nevertheless, | believe it is time we had an adult conversation about PISA. 


Are there positive things one could say about PISA? Supporters would say absolutely. They 
would argue: 


v 


v 


There is a need to have some way of comparing education systems in a global economy 
within a global village. 


Increasingly, with the exponential growth of a knowledge economy, human capital is the 
most important capital asset. 


This type of assessment, using a sampling of students, is positive in the sense that it is 
not ‘high stakes’ in that it has nothing to do with an individual student’s evaluation for 
advancement or promotion, being anonymous. Yet the results can inform jurisdictions on 
where changes to the curriculum, teacher in-service or professional development might 
be appropriate. 


There is data in these results that help to confirm what we believe or know about what is 
important. Information that confirms, as PISA does, that our system attracts strong 
teachers and prepares and in-services them well. This is essential in the turning out of 
students who achieve exceptionally well compared to international peers. But in addition, 
PISA confirms that the Canadian system has an almost incomparable system of equity. 


All these points have validity, particularly the last two. It is totally appropriate to have a 
mechanism that illustrates where some parts of a country are achieving greater success in one 
curriculum area over another. What can we learn from each other to help us improve? And the 
last point is an appreciation of PISA validating Canadian values of equity and social justice. 


Yet there are those who would criticize PISA. They would argue that: 


v 


There is an agenda behind PISA. The OECD is essentially about economics. Using 
PISA to channel the conversation about education through a narrow economic funnel is 
inappropriate to say the least. 


The narrow focus of the tests on numeracy, literacy and science does not give a valid 
picture of the whole school experience. Adding “financial literacy” to the areas tested in 
the future does not address the concern. 


The ranking of countries internationally, and provinces nationally, is problematic from 
many perspectives. There are so many factors that come into play when looking at the 
reasons why some finish high and some finish low. Should we just believe that the 


teachers and administrators in what is considered “low performing’ jurisdictions are all 
lazy and incompetent? That examples of excellence don’t happen there? Isn’t there a 
risk of stigmatizing these educators based on PISA results? 


Vv If we must compare jurisdictions, then let’s use fair comparisons, apples to apples. 
Including Chinese “economies” in the ranking seems odd. Other comparisons are at 
least questionable. 


When the first PISA tests were written in 2000, Canada ranked 6" in mathematical literacy. 
The following did not participate: 

¥ Shanghai-China (a city) 

¥ Singapore (a city state) 

v¥ Hong Kong-China ( a ‘special’ administrative region of China) 

v¥ Chinese Taipei 

v Macao-China (a ‘special’ administrative region of China) 


These all ranked ahead of Canada in 2012, when considering “statistical significance”. The 
other countries still ranked ahead of us are Korea, Japan, Liechtenstein and Switzerland. 
(Liechtenstein is the sixth smallest country in the world with a population of 35,000. It has the 
highest gross domestic product in the world). It would seem that there is some room for 
questioning the appropriateness of comparisons. 


Finally, these are undoubtedly “high stakes” tests. The degree to which PISA has informed 
government decision making: legislation, expenditures, resource allocation etc., has been 
unparalleled, and at least some of those decisions, in some jurisdictions, seem not to have been 
in the best interest of children and youth. 


In the analysis that follows, CTF Researcher Rick Riel examines more closely the OECD data 
from a Canadian perspective, expanding on this commentary. There is an argument here to 
take a closer look, continue to have an adult conversation, especially when individuals like John 
Manley, President and CEO of the Canadian Council of Chief Executives declares the results to 
be a “national emergency” for Canada. 


You can look forward to receiving additional commentary on PISA in the future as we continue 
this adult conversation. 


PISA 2012: Selected Results for Canada 
Based on the OECD Report: PISA 2012 Results in Focus, 
What the 15-year-olds know and 
what they can do with what they know 


Richard Riel, Researcher 


Top performers in reading (Level 5 or 6). 


On average across OECD countries, 8% of students are top performers in reading (Level 5 or 
6). These students can handle texts that are unfamiliar in either form or content and can 
conduct fine-grained analyses of texts. Shanghai-China has the largest proportion of top 
performers — 25% — among all participating countries 

and economies. 


More than 15% of students in Hong Kong-China, Japan and Singapore are top performers, in 
reading as are more than 10% of students in Australia, Belgium, Canada, Finland, France, 
Ireland, Korea, Liechtenstein, New Zealand, Norway, Poland and Chinese Taipei. 


PISA Mathematics Score 


Comparison to Other OECD Countries/Economies 
e Canada’s mean score in 2012 was 518, exceeding the OECD average of 494. 


e Canada’s score exceeds 5 of the 6 other G7 countries including Germany (514), France 
(495), the United Kingdom (494), Italy (485), and the United States (481); and is second 
in the G7 behind only Japan (536). 


e Canada’s score also exceeded the mean score in countries including Belgium (515), 
Austria (506), Australia (504), New Zealand (500), Norway (489), Spain (484), and 
Sweden (478). 


e Canada trailed Shanghai-China which is reported to have the highest mean at 613. 


e Although Canada ranks 13" among the 65 countries and economics, 3 of those listed 
above Canada are various economies within the same country of China, and possibly 4 
if we were to include Chinese Taipei. 


e Other than Japan, notable countries that Canada trailed in 2012 include Switzerland 
(531), the Netherlands (523) and Finland (519). 


Share of Low Achievers in Mathematics (Below Level 2) 
e Canada’s share is reported to be 13.8%, well below the OECD average of 23.1% 


e Canada’s share exceeds Japan (11.1%) but falls below the 5 remaining G7 countries, 
which experienced shares ranging from 25.8% in the USA to 17.7% in Germany. 


e The Netherlands had a slightly higher share (14.8%) than Canada despite having a 
higher mean score (523). 


e Shanghai-China experienced the lowest share at 3.8%. 


Share of top performers in mathematics (Level 5 or 6) 


e 16.4% of Canadians tested are reported to be top performers in math, exceeding the 
12.6% OECD average. 


e Canada’s share ranked third among G7 countries behind Japan (23.7%) and Germany 
(17.5%), but significantly above the USA (8.8%). Shanghai-China had the highest 
reported share at 55.4%. 


Change in Math Scores from a Country’s/Economy’s Earliest Participation in PISA to 
PISA 2012 


e Canada’s mean PISA score for math dropped by 1.4 points on an average annual basis, 
while the OECD average dropped by 0.3 points. 


e G7 countries that experienced an average annual increase in mean math score points 
included Italy (2.7), Germany (1.4), Japan (0.4), and the USA (0.3), 


e France(-1.5) was the only G7 country whose annualized mean math score points 
change declined by more than Canada, while United Kingdom (-0.3) had a smaller 
decline. 


Countries/economies whose mean score is NOT statistically significantly different from 
Canada’s Score 


e Canada’s score is not significantly different, from the Netherlands, Estonia, Finland, 
Poland, Belgium, Germany and Viet Nam, which are all statistically significantly above 
the OECD average. 


Countries whose PISA Math Scores Were Above Average in 2003 But Deteriorated 
in 2012 


e Some of the countries whose performance deteriorated more than Canada in 2012 
following above OECD average PISA performance in 2003 include the Netherlands, 
Denmark, Australia, New Zealand, Finland and Sweden. 


PISA Reading Scores 


e Canada’s mean reading score in PISA 2012 was 523, exceeding the OECD average 
of 496. 


e Canada’s 2012 score trailed Japan (538), but exceeded the remaining G7 countries 
of Germany (508), France (505), the United Kingdom (499), the USA (498), and Italy 
(490). 


e Canada’s mean score for reading declined by an average annualized change of 0.9 
points from its earliest participation in PISA to the 2012 PISA, compared to an 
average annual increase of 0.3 for the OECD average. 


e Canada incurred the most significant average annualized decline among the G7 
countries from their earliest participation in PISA to the 2012 PISA, with the USA 
being the only other G7 country to experience an annualized decline in its mean 
score (-0.3). 


PISA Science Scores 


e Canada’s mean science score in PISA 2012 was 525, exceeding the OECD average 
of 501. 


e Canada’s score trailed Japan (547), but exceeded the remaining G7 countries 
including the G7 low in Italy (494). 


e Canada’s mean score for sciences declined by an average annualized change of 1.5 
points from its earliest participation in PISA to the 2012 PISA, compared to an 
average annual increase of 0.5 points for the OECD average. 


e Italy (3.0) had the highest increase in science scores among the G7 countries from 
its earliest participation in PISA to the 2012 PISA, and the United Kingdom was the 
only other G7 country to experience a decline in score (-0.1). 


Excellence through Equity: Giving Every Student the Chance to Succeed 


What the data tell us 


Performance and equity 


Canada combined high levels of performance with equity in education opportunities as 
assessed in PISA 2012. The following countries did the same: Australia, Estonia, 
Finland, Hong Kong-China, Japan, Korea, Liechtenstein and Macao-China. 


Canada is reported to have had an above-average performance in mathematics and 
above-average equity in education opportunities. Japan was the only other G7 country 
for which this is the case. In comparison, the United States had below-average math 
performance and below average equity in education opportunities. 


The strength of the relationship between student performance in math and socio- 
economic status in Canada is weak, falling below the OECD average, while France was 
the only G7 country for which the relationship is above the OECD average. 


Student’s Engagement, Drive and Self-Beliefs 


35%of Canadian students in PISA 2012 reported having skipped classes or days of 
school, resulting in reduction of 29 score points. In comparison, the OECD average was 
25% and accounted for an average reduction of 37 score points. 


Canada’s score-points were raised 37 per unit of the index of openness to problem 
solving, exceeding the OECD average of 31 points. 


The change in Canada’s score-points associated with mathematics self-efficiency was 
an increase of 47, falling below the OECD average of 49. 


60% of students in Canada reported in PISA 2012 that their teachers often present 
problems for which there is no immediately obvious way of arriving at a solution, and 
66% reported that their teachers often present them with problems that require at least 
one week to complete. Canada is therefore more successful in this regard compared to 
the OECD average which is reported to have shares of 47% and 17% respectively. 


The percentage of students in PISA 2012 who reported being happy at school was 
above the OECD average in Canada, ranking fourth among G7 countries behind Japan, 
the United Kingdom and France. 


Teachers’ salaries and mathematics performance 


When examining teachers’ salaries relative to GDP per capita, compared to the mean 
mathematics score, Canada’s performance in math is above average among countries 
and economies with per capita GDP over 20,000$US. 


The United States, France, Italy and the United Kingdom are G7 countries that had both 
lower salaries relative to per capita GDP and lower mean mathematics scores than 
Canada in PISA 2012. 


With respect to teachers’ salaries, the OECD states the following: 


“The quality of a school cannot exceed the quality of its teachers and principals. 
Countries that have improved their performance in PISA, like Brazil, Colombia, Estonia, 
Israel, Japan and Poland, for example, have established policies to improve the quality 
of their teaching staff by either adding to the requirements to earn a teaching license, 
providing incentives for high-achieving students to enter the profession, increasing 
salaries to make the profession more attractive and to retain more teachers, or by 
offering incentives for teachers to engage in in-service teacher-training programs. While 
paying teachers well is only part of the equation, higher salaries can help school 
systems to attract the best candidates to the teaching profession. PISA results show 
that, among countries and economies whose per capita GDP is more than USD 20 000, 
high-performing school systems tend to pay more to teachers relative to their national 
income per capita.” 


